Objectives

This notebook will demonstrate how to:

  • Identify when Gene Set Variation Analysis (GSVA) is well-suited for an analysis
  • Perform GSVA on transformed RNA-seq data with the GSVA package
  • Explore the dependence of GSVA scores on gene set size with random gene sets

So far every pathway analysis method we’ve covered relies on some information about groups of samples in our data. For over-representation analysis (ORA), we created gene sets from two different two group comparisons. In the Gene Set Enrichment Analysis (GSEA) example, we used statistics from a differential gene expression (DGE) analysis where we compared MYCN amplified cell lines to non-amplified cell lines; we needed that amplification status information.

What if we’re less sure about groups in our data or we want to analyze our data in a more unsupervised manner?

In this notebook we will cover a method called Gene Set Variation Analysis (GSVA) (Hänzelmann et al. 2013) that allows us to calculate gene set or pathway scores on a per-sample basis.

We like this quote from the GSVA paper (Hänzelmann et al. 2013) to set the stage:

While [gene set enrichment] methods are generally regarded as end points of a bioinformatic analysis, GSVA constitutes a starting point to build pathway-centric models of biology.

Rather than contextualizing some results you already have from another analysis like DGE, GSVA is designed to provide an estimate of pathway variation for each of the samples in an experiment. Note that these scores will depend on the samples included in the dataset when you run GSVA; if you added more samples and reran GSVA, you would expect the scores to change.

Set up

Libraries

# Pipes
library(magrittr)
# Gene Set Variation Analysis
library(GSVA)

Directories and files

Directories

# We have some medulloblastoma data from the OpenPBTA project that we've 
# prepared ahead of time
input_dir <- file.path("data", "open-pbta")

# Create a directory specifically for the results using this dataset
output_dir <- file.path("results", "open-pbta") 
if (!dir.exists(output_dir)) {
  dir.create(output_dir, recursive = TRUE)
}

Input

We have VST transformed RNA-seq data, annotated with gene symbols, that has been collapsed such that there are no duplicated gene identifiers (see setup).

rnaseq_file <- file.path(input_dir, "medulloblastoma_vst_collapsed.tsv")

Output

gsva_results_file <- file.path(output_dir, "medulloblastoma_gsva_results.tsv")

Gene sets

The function that we will use to run GSVA wants the gene sets to be in a list, rather than a tidy data frame that we used with clusterProfiler (although it does accept other formats).

We’re going to take this opportunity to introduce a different format that gene sets are often distributed in called GMT (Gene Matrix Transposed).

We’re going to read in the Hallmark collection file directly from MSigDB, rather than using msigdbr like we did in earlier notebooks.

The RNA-seq data uses gene symbols, so we need gene sets that use gene symbols, too.

# R can often read in data from a URL
hallmarks_url <- "https://data.broadinstitute.org/gsea-msigdb/msigdb/release/7.1/h.all.v7.1.symbols.gmt"

# QuSAGE is another pathway analysis method, the qusage package has a function
# for reading GMT files and turning them into a list
hallmarks_list <- qusage::read.gmt(hallmarks_url)

What does this list look like?

head(hallmarks_list)

RNA-seq data

We have VST transformed RNA-seq data, which is on a log2-like scale. These data are from the Open Pediatric Brain Tumor Atlas (OpenPBTA) OpenPBTA is a collaborative project organized by the CCDL and the Center for Data-Driven Discovery in Biomedicine (D3b) at the Children’s Hospital of Philadelphia conducted openly on GitHub.

You can read more about the project here.

We’re only working with the medulloblastoma samples in this example.

rnaseq_df <- readr::read_tsv(rnaseq_file)

── Column specification ──────────────────────────────────────────────────────────────────────────────────────────────────────
cols(
  .default = col_double(),
  gene_symbol = col_character()
)
ℹ Use `spec()` for the full column specifications.
# What does the RNA-seq data frame look like?
rnaseq_df[1:5, 1:5]

For GSVA, we need a matrix.

rnaseq_mat <- rnaseq_df %>%
  tibble::column_to_rownames("gene_symbol") %>%
  as.matrix()

Note: If we had duplicate gene symbols here, we couldn’t set them as rownames.

GSVA

Figure 1 from Hänzelmann et al. (2013).

You may notice that GSVA has some commonalities with GSEA. Rather than ranking genes based on some statistic we selected ahead of time, GSVA fits a model and ranks genes based on their expression level relative to the sample distribution. This is a way of asking if a gene i is highly or lowly expressed in a sample j in the context of this experiment and ranking accordingly (Hänzelmann et al. 2013). The pathway-level score calculated is a way of asking how genes within a gene set vary as compared to genes that are outside of that gene set (Malhotra. 2018). (This is sometimes called a competitive test in gene set enrichment literature.) The intuition here is that we will get pathway-level scores for each sample that indicate if genes in a pathway vary concordantly in one direction (overexpressed or underexpressed relative to the overall population) (Hänzelmann et al. 2013).

The output is a gene set by sample matrix of GSVA scores.

Perform GSVA

gsva_results <- gsva(rnaseq_mat, 
                     hallmarks_list,
                     method = "gsva",
                     # Appropriate for our transformed data on log2-like scale
                     kcdf = "Gaussian",  
                     # Minimum gene set size
                     min.sz = 15, 
                     # Maximum gene set size
                     max.sz = 500,
                     # Compute Gaussian-distributed scores
                     mx.diff = TRUE)
945 genes with constant expression values throuhgout the samples.Since argument method!="ssgsea", genes with constant expression values are discarded.
Estimating GSVA scores for 50 gene sets.
Estimating ECDFs with Gaussian kernels

  |                                                                                                                          
  |                                                                                                                    |   0%
  |                                                                                                                          
  |==                                                                                                                  |   2%
  |                                                                                                                          
  |=====                                                                                                               |   4%
  |                                                                                                                          
  |=======                                                                                                             |   6%
  |                                                                                                                          
  |=========                                                                                                           |   8%
  |                                                                                                                          
  |============                                                                                                        |  10%
  |                                                                                                                          
  |==============                                                                                                      |  12%
  |                                                                                                                          
  |================                                                                                                    |  14%
  |                                                                                                                          
  |===================                                                                                                 |  16%
  |                                                                                                                          
  |=====================                                                                                               |  18%
  |                                                                                                                          
  |=======================                                                                                             |  20%
  |                                                                                                                          
  |==========================                                                                                          |  22%
  |                                                                                                                          
  |============================                                                                                        |  24%
  |                                                                                                                          
  |==============================                                                                                      |  26%
  |                                                                                                                          
  |================================                                                                                    |  28%
  |                                                                                                                          
  |===================================                                                                                 |  30%
  |                                                                                                                          
  |=====================================                                                                               |  32%
  |                                                                                                                          
  |=======================================                                                                             |  34%
  |                                                                                                                          
  |==========================================                                                                          |  36%
  |                                                                                                                          
  |============================================                                                                        |  38%
  |                                                                                                                          
  |==============================================                                                                      |  40%
  |                                                                                                                          
  |=================================================                                                                   |  42%
  |                                                                                                                          
  |===================================================                                                                 |  44%
  |                                                                                                                          
  |=====================================================                                                               |  46%
  |                                                                                                                          
  |========================================================                                                            |  48%
  |                                                                                                                          
  |==========================================================                                                          |  50%
  |                                                                                                                          
  |============================================================                                                        |  52%
  |                                                                                                                          
  |===============================================================                                                     |  54%
  |                                                                                                                          
  |=================================================================                                                   |  56%
  |                                                                                                                          
  |===================================================================                                                 |  58%
  |                                                                                                                          
  |======================================================================                                              |  60%
  |                                                                                                                          
  |========================================================================                                            |  62%
  |                                                                                                                          
  |==========================================================================                                          |  64%
  |                                                                                                                          
  |=============================================================================                                       |  66%
  |                                                                                                                          
  |===============================================================================                                     |  68%
  |                                                                                                                          
  |=================================================================================                                   |  70%
  |                                                                                                                          
  |====================================================================================                                |  72%
  |                                                                                                                          
  |======================================================================================                              |  74%
  |                                                                                                                          
  |========================================================================================                            |  76%
  |                                                                                                                          
  |==========================================================================================                          |  78%
  |                                                                                                                          
  |=============================================================================================                       |  80%
  |                                                                                                                          
  |===============================================================================================                     |  82%
  |                                                                                                                          
  |=================================================================================================                   |  84%
  |                                                                                                                          
  |====================================================================================================                |  86%
  |                                                                                                                          
  |======================================================================================================              |  88%
  |                                                                                                                          
  |========================================================================================================            |  90%
  |                                                                                                                          
  |===========================================================================================================         |  92%
  |                                                                                                                          
  |=============================================================================================================       |  94%
  |                                                                                                                          
  |===============================================================================================================     |  96%
  |                                                                                                                          
  |==================================================================================================================  |  98%
  |                                                                                                                          
  |====================================================================================================================| 100%

Note: the gsva() documentation says we can use kcdf = "Gaussian" if we had RNA-seq log-CPMs, log-RPKMs or log-TPMs, but we would use kcdf = "Poisson" on integer counts.

# Let's explore what the output of gsva() looks like
gsva_results[1:5, 1:5]
                                    BS_09Z7TC35 BS_1AYRM596 BS_1BWP5MCT BS_1QXEC43H BS_1TWCV047
HALLMARK_TNFA_SIGNALING_VIA_NFKB    -0.44911439   0.5208667  -0.5609193  0.32300630  -0.1468081
HALLMARK_HYPOXIA                    -0.38297104   0.2436910  -0.5058759  0.36247083  -0.2971559
HALLMARK_CHOLESTEROL_HOMEOSTASIS    -0.26534735   0.1054224  -0.5180933  0.32418657  -0.4561386
HALLMARK_MITOTIC_SPINDLE             0.12727006   0.2339489  -0.4338076 -0.07023068  -0.1861483
HALLMARK_WNT_BETA_CATENIN_SIGNALING -0.09287646   0.1602124  -0.4427594  0.41372523  -0.1152437

A note on gene set size

Often the scores of gene set enrichment methods are not comparable between gene sets of different sizes. Let’s do an experiment using randomly generated gene sets to explore this idea a bit more.

We need to get a collection of all possible genes we will sample from to create random gene sets. Because we’re doing some random sampling, we need to set a seed for this to be reproducible.

# Use all the gene symbols in the dataset as the pool of possible genes
all_genes <- rownames(rnaseq_mat)

# Set a seed for reproducibility
set.seed(2020)

Our minimum gene set size earlier was 15 genes and our maximum gene set size was 500 genes. We’ll use the same minimum and maximum values for our random gene sets and some values in between.

# Make a list of integers that indicate the random gene set sizes
gene_set_size <- list(15, 25, 50, 100, 250, 500)

For each gene set size, we will generate 100 random gene sets.

# Set number of replicates
nreps <- 100
# Generate 100 random gene sets of each size
random_gene_sets <- 
  purrr::map(
    rep(gene_set_size, nreps),  # Repeat gene sizes so we run `nreps` times 
    # Sample the vector of all genes, choosing the number of items specified
    # in the element of gene set size
    ~ base::sample(x = all_genes,
                   size = .x)
  )

The Hallmarks list we used earlier stored the gene set names as the name of the list, so let’s add names to our random gene sets that indicate what size they are and so gsva() doesn’t get upset.

# We will include the size of the gene set in the gene set name 
# Start by taking the length of each pathway and appending "pathway_" to that
# number
lengths_vector <- random_gene_sets %>%
  # Get the length of each gene set (number of genes)
  purrr::map(~ length(.x)) %>%
  # Make it "pathway_<gene set size>"
  purrr::map(~ paste0("pathway_", .x)) %>%
  # Return a vector
  purrr::flatten_chr()

# Add the names in lengths_vector to the list - "pathway_<gene set size>"
random_gene_sets <- random_gene_sets %>% 
  # make.names() appends a "version" if something is not unique
  purrr::set_names(nm = make.names(lengths_vector, unique = TRUE))

Run GSVA on our dataset with the same parameters as before, but now with random gene sets.

random_gsva_results <- gsva(rnaseq_mat, 
                            random_gene_sets,
                            method = "gsva",
                            # Appropriate for our transformed data on 
                            # log2-like scale
                            kcdf = "Gaussian",  
                            # Minimum gene set size
                            min.sz = 15, 
                            # Maximum gene set size
                            max.sz = 500,
                            # Compute Gaussian-distributed scores
                            mx.diff = TRUE)                            
945 genes with constant expression values throuhgout the samples.Since argument method!="ssgsea", genes with constant expression values are discarded.
Estimating GSVA scores for 579 gene sets.
Estimating ECDFs with Gaussian kernels

  |                                                                                                                          
  |                                                                                                                    |   0%
  |                                                                                                                          
  |=                                                                                                                   |   1%
  |                                                                                                                          
  |==                                                                                                                  |   1%
  |                                                                                                                          
  |==                                                                                                                  |   2%
  |                                                                                                                          
  |===                                                                                                                 |   2%
  |                                                                                                                          
  |===                                                                                                                 |   3%
  |                                                                                                                          
  |====                                                                                                                |   3%
  |                                                                                                                          
  |====                                                                                                                |   4%
  |                                                                                                                          
  |=====                                                                                                               |   4%
  |                                                                                                                          
  |=====                                                                                                               |   5%
  |                                                                                                                          
  |======                                                                                                              |   5%
  |                                                                                                                          
  |======                                                                                                              |   6%
  |                                                                                                                          
  |=======                                                                                                             |   6%
  |                                                                                                                          
  |========                                                                                                            |   7%
  |                                                                                                                          
  |=========                                                                                                           |   7%
  |                                                                                                                          
  |=========                                                                                                           |   8%
  |                                                                                                                          
  |==========                                                                                                          |   8%
  |                                                                                                                          
  |==========                                                                                                          |   9%
  |                                                                                                                          
  |===========                                                                                                         |   9%
  |                                                                                                                          
  |===========                                                                                                         |  10%
  |                                                                                                                          
  |============                                                                                                        |  10%
  |                                                                                                                          
  |============                                                                                                        |  11%
  |                                                                                                                          
  |=============                                                                                                       |  11%
  |                                                                                                                          
  |=============                                                                                                       |  12%
  |                                                                                                                          
  |==============                                                                                                      |  12%
  |                                                                                                                          
  |===============                                                                                                     |  13%
  |                                                                                                                          
  |================                                                                                                    |  13%
  |                                                                                                                          
  |================                                                                                                    |  14%
  |                                                                                                                          
  |=================                                                                                                   |  14%
  |                                                                                                                          
  |=================                                                                                                   |  15%
  |                                                                                                                          
  |==================                                                                                                  |  15%
  |                                                                                                                          
  |==================                                                                                                  |  16%
  |                                                                                                                          
  |===================                                                                                                 |  16%
  |                                                                                                                          
  |===================                                                                                                 |  17%
  |                                                                                                                          
  |====================                                                                                                |  17%
  |                                                                                                                          
  |====================                                                                                                |  18%
  |                                                                                                                          
  |=====================                                                                                               |  18%
  |                                                                                                                          
  |======================                                                                                              |  19%
  |                                                                                                                          
  |=======================                                                                                             |  20%
  |                                                                                                                          
  |========================                                                                                            |  20%
  |                                                                                                                          
  |========================                                                                                            |  21%
  |                                                                                                                          
  |=========================                                                                                           |  21%
  |                                                                                                                          
  |=========================                                                                                           |  22%
  |                                                                                                                          
  |==========================                                                                                          |  22%
  |                                                                                                                          
  |==========================                                                                                          |  23%
  |                                                                                                                          
  |===========================                                                                                         |  23%
  |                                                                                                                          
  |===========================                                                                                         |  24%
  |                                                                                                                          
  |============================                                                                                        |  24%
  |                                                                                                                          
  |============================                                                                                        |  25%
  |                                                                                                                          
  |=============================                                                                                       |  25%
  |                                                                                                                          
  |==============================                                                                                      |  26%
  |                                                                                                                          
  |===============================                                                                                     |  26%
  |                                                                                                                          
  |===============================                                                                                     |  27%
  |                                                                                                                          
  |================================                                                                                    |  27%
  |                                                                                                                          
  |================================                                                                                    |  28%
  |                                                                                                                          
  |=================================                                                                                   |  28%
  |                                                                                                                          
  |=================================                                                                                   |  29%
  |                                                                                                                          
  |==================================                                                                                  |  29%
  |                                                                                                                          
  |==================================                                                                                  |  30%
  |                                                                                                                          
  |===================================                                                                                 |  30%
  |                                                                                                                          
  |===================================                                                                                 |  31%
  |                                                                                                                          
  |====================================                                                                                |  31%
  |                                                                                                                          
  |=====================================                                                                               |  32%
  |                                                                                                                          
  |======================================                                                                              |  32%
  |                                                                                                                          
  |======================================                                                                              |  33%
  |                                                                                                                          
  |=======================================                                                                             |  33%
  |                                                                                                                          
  |=======================================                                                                             |  34%
  |                                                                                                                          
  |========================================                                                                            |  34%
  |                                                                                                                          
  |========================================                                                                            |  35%
  |                                                                                                                          
  |=========================================                                                                           |  35%
  |                                                                                                                          
  |=========================================                                                                           |  36%
  |                                                                                                                          
  |==========================================                                                                          |  36%
  |                                                                                                                          
  |==========================================                                                                          |  37%
  |                                                                                                                          
  |===========================================                                                                         |  37%
  |                                                                                                                          
  |============================================                                                                        |  38%
  |                                                                                                                          
  |=============================================                                                                       |  39%
  |                                                                                                                          
  |==============================================                                                                      |  39%
  |                                                                                                                          
  |==============================================                                                                      |  40%
  |                                                                                                                          
  |===============================================                                                                     |  40%
  |                                                                                                                          
  |===============================================                                                                     |  41%
  |                                                                                                                          
  |================================================                                                                    |  41%
  |                                                                                                                          
  |================================================                                                                    |  42%
  |                                                                                                                          
  |=================================================                                                                   |  42%
  |                                                                                                                          
  |=================================================                                                                   |  43%
  |                                                                                                                          
  |==================================================                                                                  |  43%
  |                                                                                                                          
  |==================================================                                                                  |  44%
  |                                                                                                                          
  |===================================================                                                                 |  44%
  |                                                                                                                          
  |====================================================                                                                |  45%
  |                                                                                                                          
  |=====================================================                                                               |  45%
  |                                                                                                                          
  |=====================================================                                                               |  46%
  |                                                                                                                          
  |======================================================                                                              |  46%
  |                                                                                                                          
  |======================================================                                                              |  47%
  |                                                                                                                          
  |=======================================================                                                             |  47%
  |                                                                                                                          
  |=======================================================                                                             |  48%
  |                                                                                                                          
  |========================================================                                                            |  48%
  |                                                                                                                          
  |========================================================                                                            |  49%
  |                                                                                                                          
  |=========================================================                                                           |  49%
  |                                                                                                                          
  |=========================================================                                                           |  50%
  |                                                                                                                          
  |==========================================================                                                          |  50%
  |                                                                                                                          
  |===========================================================                                                         |  50%
  |                                                                                                                          
  |===========================================================                                                         |  51%
  |                                                                                                                          
  |============================================================                                                        |  51%
  |                                                                                                                          
  |============================================================                                                        |  52%
  |                                                                                                                          
  |=============================================================                                                       |  52%
  |                                                                                                                          
  |=============================================================                                                       |  53%
  |                                                                                                                          
  |==============================================================                                                      |  53%
  |                                                                                                                          
  |==============================================================                                                      |  54%
  |                                                                                                                          
  |===============================================================                                                     |  54%
  |                                                                                                                          
  |===============================================================                                                     |  55%
  |                                                                                                                          
  |================================================================                                                    |  55%
  |                                                                                                                          
  |=================================================================                                                   |  56%
  |                                                                                                                          
  |==================================================================                                                  |  56%
  |                                                                                                                          
  |==================================================================                                                  |  57%
  |                                                                                                                          
  |===================================================================                                                 |  57%
  |                                                                                                                          
  |===================================================================                                                 |  58%
  |                                                                                                                          
  |====================================================================                                                |  58%
  |                                                                                                                          
  |====================================================================                                                |  59%
  |                                                                                                                          
  |=====================================================================                                               |  59%
  |                                                                                                                          
  |=====================================================================                                               |  60%
  |                                                                                                                          
  |======================================================================                                              |  60%
  |                                                                                                                          
  |======================================================================                                              |  61%
  |                                                                                                                          
  |=======================================================================                                             |  61%
  |                                                                                                                          
  |========================================================================                                            |  62%
  |                                                                                                                          
  |=========================================================================                                           |  63%
  |                                                                                                                          
  |==========================================================================                                          |  63%
  |                                                                                                                          
  |==========================================================================                                          |  64%
  |                                                                                                                          
  |===========================================================================                                         |  64%
  |                                                                                                                          
  |===========================================================================                                         |  65%
  |                                                                                                                          
  |============================================================================                                        |  65%
  |                                                                                                                          
  |============================================================================                                        |  66%
  |                                                                                                                          
  |=============================================================================                                       |  66%
  |                                                                                                                          
  |=============================================================================                                       |  67%
  |                                                                                                                          
  |==============================================================================                                      |  67%
  |                                                                                                                          
  |==============================================================================                                      |  68%
  |                                                                                                                          
  |===============================================================================                                     |  68%
  |                                                                                                                          
  |================================================================================                                    |  69%
  |                                                                                                                          
  |=================================================================================                                   |  69%
  |                                                                                                                          
  |=================================================================================                                   |  70%
  |                                                                                                                          
  |==================================================================================                                  |  70%
  |                                                                                                                          
  |==================================================================================                                  |  71%
  |                                                                                                                          
  |===================================================================================                                 |  71%
  |                                                                                                                          
  |===================================================================================                                 |  72%
  |                                                                                                                          
  |====================================================================================                                |  72%
  |                                                                                                                          
  |====================================================================================                                |  73%
  |                                                                                                                          
  |=====================================================================================                               |  73%
  |                                                                                                                          
  |=====================================================================================                               |  74%
  |                                                                                                                          
  |======================================================================================                              |  74%
  |                                                                                                                          
  |=======================================================================================                             |  75%
  |                                                                                                                          
  |========================================================================================                            |  75%
  |                                                                                                                          
  |========================================================================================                            |  76%
  |                                                                                                                          
  |=========================================================================================                           |  76%
  |                                                                                                                          
  |=========================================================================================                           |  77%
  |                                                                                                                          
  |==========================================================================================                          |  77%
  |                                                                                                                          
  |==========================================================================================                          |  78%
  |                                                                                                                          
  |===========================================================================================                         |  78%
  |                                                                                                                          
  |===========================================================================================                         |  79%
  |                                                                                                                          
  |============================================================================================                        |  79%
  |                                                                                                                          
  |============================================================================================                        |  80%
  |                                                                                                                          
  |=============================================================================================                       |  80%
  |                                                                                                                          
  |==============================================================================================                      |  81%
  |                                                                                                                          
  |===============================================================================================                     |  82%
  |                                                                                                                          
  |================================================================================================                    |  82%
  |                                                                                                                          
  |================================================================================================                    |  83%
  |                                                                                                                          
  |=================================================================================================                   |  83%
  |                                                                                                                          
  |=================================================================================================                   |  84%
  |                                                                                                                          
  |==================================================================================================                  |  84%
  |                                                                                                                          
  |==================================================================================================                  |  85%
  |                                                                                                                          
  |===================================================================================================                 |  85%
  |                                                                                                                          
  |===================================================================================================                 |  86%
  |                                                                                                                          
  |====================================================================================================                |  86%
  |                                                                                                                          
  |====================================================================================================                |  87%
  |                                                                                                                          
  |=====================================================================================================               |  87%
  |                                                                                                                          
  |======================================================================================================              |  88%
  |                                                                                                                          
  |=======================================================================================================             |  88%
  |                                                                                                                          
  |=======================================================================================================             |  89%
  |                                                                                                                          
  |========================================================================================================            |  89%
  |                                                                                                                          
  |========================================================================================================            |  90%
  |                                                                                                                          
  |=========================================================================================================           |  90%
  |                                                                                                                          
  |=========================================================================================================           |  91%
  |                                                                                                                          
  |==========================================================================================================          |  91%
  |                                                                                                                          
  |==========================================================================================================          |  92%
  |                                                                                                                          
  |===========================================================================================================         |  92%
  |                                                                                                                          
  |===========================================================================================================         |  93%
  |                                                                                                                          
  |============================================================================================================        |  93%
  |                                                                                                                          
  |=============================================================================================================       |  94%
  |                                                                                                                          
  |==============================================================================================================      |  94%
  |                                                                                                                          
  |==============================================================================================================      |  95%
  |                                                                                                                          
  |===============================================================================================================     |  95%
  |                                                                                                                          
  |===============================================================================================================     |  96%
  |                                                                                                                          
  |================================================================================================================    |  96%
  |                                                                                                                          
  |================================================================================================================    |  97%
  |                                                                                                                          
  |=================================================================================================================   |  97%
  |                                                                                                                          
  |=================================================================================================================   |  98%
  |                                                                                                                          
  |==================================================================================================================  |  98%
  |                                                                                                                          
  |==================================================================================================================  |  99%
  |                                                                                                                          
  |=================================================================================================================== |  99%
  |                                                                                                                          
  |====================================================================================================================| 100%

Now let’s make a plot to look at the distribution of scores from random gene sets. First we need to get this data in an appropriate format for ggplot2.

# The random results are a matrix
random_long_df <- random_gsva_results %>%
  data.frame() %>%
  # Gene set names are rownames
  tibble::rownames_to_column("gene_set") %>%
  # Get into long format
  tidyr::pivot_longer(cols = -gene_set, 
                      names_to = "Kids_First_Biospecimen_ID",
                      values_to = "gsva_score") %>%
  # Remove the .version added by make.names()
  dplyr::mutate(gene_set = stringr::str_remove(gene_set, "\\..*")) %>%
  # Add a column that keeps track of the gene set size
  dplyr::mutate(gene_set_size = stringr::word(gene_set, 2, sep = "_")) %>%
  # We want to plot smallest no. genes -> largest no. genes
  dplyr::mutate(gene_set_size = factor(gene_set_size, 
                                       levels = c(15, 25, 50, 100, 250, 500)))

Let’s make a violin plot so we can look at the distribution of scores by gene set size.

# Violin plot comparing GSVA scores of different random gene set sizes
random_long_df %>%
  ggplot2::ggplot(ggplot2::aes(x = gene_set_size, 
                               y = gsva_score)) +
  # Make a violin plot that's a pretty blue!
  ggplot2::geom_violin(fill = "#99CCFF", alpha = 0.5) +
  # Add a point with the mean value
  ggplot2::stat_summary(
    geom = "point",
    fun = "mean",
    # Change the aesthetics of the points
    size = 3,
    color = "#0066CC",
    shape = 18
  ) +
  # Flip the axes
  ggplot2::coord_flip() +
  ggplot2::labs(title = "Random Gene Set GSVA Scores",
                x = "gene set size",
                y = "GSVA score") +
  ggplot2::theme_bw()

What do you notice about these distributions? How might you use this information to inform your interpretation of GSVA scores?

How can you use these scores?

If you did have groups information for your samples, you could test for pathway scores differences between groups. Here’s an example of that from the OpenPBTA project itself!

You can also visualize this matrix in a heatmap. Here’s a figure from the OpenPBTA project, where the middle panel is a heatmap of GSVA scores that were significantly different between histologies.

Code here.

Write results to file

gsva_results %>%
  as.data.frame() %>%
  tibble::rownames_to_column("pathway") %>%
  readr::write_tsv(file = gsva_results_file)

Session Info

sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8   
 [6] LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C           LC_TELEPHONE=C        
[11] LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
[1] GSVA_1.38.2    magrittr_2.0.1

loaded via a namespace (and not attached):
  [1] fgsea_1.16.0                colorspace_2.0-0            ellipsis_0.3.1              estimability_1.3           
  [5] qvalue_2.22.0               XVector_0.30.0              GenomicRanges_1.42.0        rstudioapi_0.13            
  [9] farver_2.0.3                graphlayouts_0.7.1          ggrepel_0.9.1               bit64_4.0.5                
 [13] fansi_0.4.2                 AnnotationDbi_1.52.0        mvtnorm_1.1-1               scatterpie_0.1.5           
 [17] splines_4.0.3               cachem_1.0.1                GOSemSim_2.16.1             knitr_1.30                 
 [21] polyclip_1.10-0             jsonlite_1.7.2              annotate_1.68.0             GO.db_3.12.1               
 [25] graph_1.68.0                ggforce_0.3.2               BiocManager_1.30.10         readr_1.4.0                
 [29] compiler_4.0.3              httr_1.4.2                  rvcheck_0.1.8               emmeans_1.5.3              
 [33] assertthat_0.2.1            Matrix_1.3-2                fastmap_1.1.0               cli_2.2.0                  
 [37] limma_3.46.0                tweenr_1.0.1                htmltools_0.5.1.1           tools_4.0.3                
 [41] igraph_1.2.6                coda_0.19-4                 gtable_0.3.0                glue_1.4.2                 
 [45] GenomeInfoDbData_1.2.4      reshape2_1.4.4              DO.db_2.9                   dplyr_1.0.3                
 [49] fastmatch_1.1-0             Rcpp_1.0.6                  enrichplot_1.10.2           Biobase_2.50.0             
 [53] vctrs_0.3.6                 nlme_3.1-151                ggraph_2.0.4                xfun_0.20                  
 [57] stringr_1.4.0               qusage_2.24.0               lifecycle_0.2.0             renv_0.12.5-2              
 [61] XML_3.99-0.5                DOSE_3.16.0                 zlibbioc_1.36.0             MASS_7.3-53                
 [65] scales_1.1.1                tidygraph_1.2.0             hms_1.0.0                   MatrixGenerics_1.2.0       
 [69] parallel_4.0.3              SummarizedExperiment_1.20.0 RColorBrewer_1.1-2          yaml_2.2.1                 
 [73] memoise_1.1.0               gridExtra_2.3               ggplot2_3.3.3               stringi_1.5.3              
 [77] RSQLite_2.2.3               S4Vectors_0.28.1            BiocGenerics_0.36.0         BiocParallel_1.24.1        
 [81] GenomeInfoDb_1.26.2         rlang_0.4.10                pkgconfig_2.0.3             matrixStats_0.57.0         
 [85] bitops_1.0-6                evaluate_0.14               lattice_0.20-41             purrr_0.3.4                
 [89] labeling_0.4.2              cowplot_1.1.1               shadowtext_0.0.7            bit_4.0.4                  
 [93] tidyselect_1.1.0            GSEABase_1.52.1             plyr_1.8.6                  R6_2.5.0                   
 [97] IRanges_2.24.1              fftw_1.0-6                  generics_0.1.0              DelayedArray_0.16.2        
[101] DBI_1.1.1                   pillar_1.4.7                RCurl_1.98-1.2              tibble_3.0.5               
[105] crayon_1.3.4                rmarkdown_2.6               viridis_0.5.1               grid_4.0.3                 
[109] data.table_1.13.6           blob_1.2.1                  digest_0.6.27               xtable_1.8-4               
[113] tidyr_1.1.2                 stats4_4.0.3                munsell_0.5.0               viridisLite_0.3.0          
LS0tCnRpdGxlOiAiUGF0aHdheSBhbmFseXNpczogR2VuZSBTZXQgVmFyaWF0aW9uIEFuYWx5c2lzIChHU1ZBKSIKb3V0cHV0OiAKICBodG1sX25vdGVib29rOgogICAgdG9jOiB0cnVlCiAgICB0b2NfZmxvYXQ6IHRydWUKYXV0aG9yOiBDQ0RMIGZvciBBTFNGCmRhdGU6IDIwMjAKLS0tCgojIyBPYmplY3RpdmVzCgpUaGlzIG5vdGVib29rIHdpbGwgZGVtb25zdHJhdGUgaG93IHRvOgoKLSBJZGVudGlmeSB3aGVuIEdlbmUgU2V0IFZhcmlhdGlvbiBBbmFseXNpcyAoR1NWQSkgaXMgd2VsbC1zdWl0ZWQgZm9yIGFuIGFuYWx5c2lzCi0gUGVyZm9ybSBHU1ZBIG9uIHRyYW5zZm9ybWVkIFJOQS1zZXEgZGF0YSB3aXRoIHRoZSBgR1NWQWAgcGFja2FnZQotIEV4cGxvcmUgdGhlIGRlcGVuZGVuY2Ugb2YgR1NWQSBzY29yZXMgb24gZ2VuZSBzZXQgc2l6ZSB3aXRoIHJhbmRvbSBnZW5lIHNldHMKCi0tLQoKU28gZmFyIGV2ZXJ5IHBhdGh3YXkgYW5hbHlzaXMgbWV0aG9kIHdlJ3ZlIGNvdmVyZWQgcmVsaWVzIG9uIHNvbWUgaW5mb3JtYXRpb24gYWJvdXQgZ3JvdXBzIG9mIHNhbXBsZXMgaW4gb3VyIGRhdGEuCkZvciBvdmVyLXJlcHJlc2VudGF0aW9uIGFuYWx5c2lzIChPUkEpLCB3ZSBjcmVhdGVkIGdlbmUgc2V0cyBmcm9tIHR3byBkaWZmZXJlbnQgdHdvIGdyb3VwIGNvbXBhcmlzb25zLgpJbiB0aGUgR2VuZSBTZXQgRW5yaWNobWVudCBBbmFseXNpcyAoR1NFQSkgZXhhbXBsZSwgd2UgdXNlZCBzdGF0aXN0aWNzIGZyb20gYSBkaWZmZXJlbnRpYWwgZ2VuZSBleHByZXNzaW9uIChER0UpIGFuYWx5c2lzIHdoZXJlIHdlIGNvbXBhcmVkIF9NWUNOXyBhbXBsaWZpZWQgY2VsbCBsaW5lcyB0byBub24tYW1wbGlmaWVkIGNlbGwgbGluZXM7IHdlIG5lZWRlZCB0aGF0IGFtcGxpZmljYXRpb24gc3RhdHVzIGluZm9ybWF0aW9uLgoKV2hhdCBpZiB3ZSdyZSBsZXNzIHN1cmUgYWJvdXQgZ3JvdXBzIGluIG91ciBkYXRhIG9yIHdlIHdhbnQgdG8gYW5hbHl6ZSBvdXIgZGF0YSBpbiBhIG1vcmUgdW5zdXBlcnZpc2VkIG1hbm5lcj8KCkluIHRoaXMgbm90ZWJvb2sgd2Ugd2lsbCBjb3ZlciBhIG1ldGhvZCBjYWxsZWQgR2VuZSBTZXQgVmFyaWF0aW9uIEFuYWx5c2lzIChHU1ZBKSAoW0jDpG56ZWxtYW5uIF9ldCBhbC5fIDIwMTNdKGh0dHBzOi8vZG9pLm9yZy8xMC4xMTg2LzE0NzEtMjEwNS0xNC03KSkgdGhhdCBhbGxvd3MgdXMgdG8gY2FsY3VsYXRlIGdlbmUgc2V0IG9yIHBhdGh3YXkgc2NvcmVzIG9uIGEgcGVyLXNhbXBsZSBiYXNpcy4KCldlIGxpa2UgdGhpcyBxdW90ZSBmcm9tIHRoZSBHU1ZBIHBhcGVyIChbSMOkbnplbG1hbm4gX2V0IGFsLl8gMjAxM10oaHR0cHM6Ly9kb2kub3JnLzEwLjExODYvMTQ3MS0yMTA1LTE0LTcpKSB0byBzZXQgdGhlIHN0YWdlOgoKPiBXaGlsZSBbZ2VuZSBzZXQgZW5yaWNobWVudF0gbWV0aG9kcyBhcmUgZ2VuZXJhbGx5IHJlZ2FyZGVkIGFzIGVuZCBwb2ludHMgb2YgYSBiaW9pbmZvcm1hdGljIGFuYWx5c2lzLCBHU1ZBIGNvbnN0aXR1dGVzIGEgc3RhcnRpbmcgcG9pbnQgdG8gYnVpbGQgcGF0aHdheS1jZW50cmljIG1vZGVscyBvZiBiaW9sb2d5LgoKUmF0aGVyIHRoYW4gY29udGV4dHVhbGl6aW5nIHNvbWUgcmVzdWx0cyB5b3UgX2FscmVhZHkgaGF2ZV8gZnJvbSBhbm90aGVyIGFuYWx5c2lzIGxpa2UgREdFLCBHU1ZBIGlzIGRlc2lnbmVkIHRvIHByb3ZpZGUgYW4gZXN0aW1hdGUgb2YgcGF0aHdheSB2YXJpYXRpb24gZm9yIGVhY2ggb2YgdGhlIHNhbXBsZXMgaW4gYW4gZXhwZXJpbWVudC4KTm90ZSB0aGF0IHRoZXNlIHNjb3JlcyB3aWxsIGRlcGVuZCBvbiB0aGUgc2FtcGxlcyBpbmNsdWRlZCBpbiB0aGUgZGF0YXNldCB3aGVuIHlvdSBydW4gR1NWQTsgaWYgeW91IGFkZGVkIG1vcmUgc2FtcGxlcyBhbmQgcmVyYW4gR1NWQSwgeW91IHdvdWxkIGV4cGVjdCB0aGUgc2NvcmVzIHRvIGNoYW5nZS4KCiMjIyMgT3RoZXIgcmVzb3VyY2VzCgoqIFtNYWxob3RyYS4gX0RlY29kaW5nIEdlbmUgU2V0IFZhcmlhdGlvbiBBbmFseXNpc18uICgyMDE4KV0oaHR0cHM6Ly90b3dhcmRzZGF0YXNjaWVuY2UuY29tL2RlY29kaW5nLWdlbmUtc2V0LXZhcmlhdGlvbi1hbmFseXNpcy04MTkzYTBjZmRhMykKCiMjIFNldCB1cAoKIyMjIExpYnJhcmllcwoKYGBge3IgbGlicmFyaWVzfQojIFBpcGVzCmxpYnJhcnkobWFncml0dHIpCiMgR2VuZSBTZXQgVmFyaWF0aW9uIEFuYWx5c2lzCmxpYnJhcnkoR1NWQSkKYGBgCgojIyMgRGlyZWN0b3JpZXMgYW5kIGZpbGVzCgojIyMjIERpcmVjdG9yaWVzCgpgYGB7ciBkaXJlY3Rvcmllc30KIyBXZSBoYXZlIHNvbWUgbWVkdWxsb2JsYXN0b21hIGRhdGEgZnJvbSB0aGUgT3BlblBCVEEgcHJvamVjdCB0aGF0IHdlJ3ZlIAojIHByZXBhcmVkIGFoZWFkIG9mIHRpbWUKaW5wdXRfZGlyIDwtIGZpbGUucGF0aCgiZGF0YSIsICJvcGVuLXBidGEiKQoKIyBDcmVhdGUgYSBkaXJlY3Rvcnkgc3BlY2lmaWNhbGx5IGZvciB0aGUgcmVzdWx0cyB1c2luZyB0aGlzIGRhdGFzZXQKb3V0cHV0X2RpciA8LSBmaWxlLnBhdGgoInJlc3VsdHMiLCAib3Blbi1wYnRhIikgCmlmICghZGlyLmV4aXN0cyhvdXRwdXRfZGlyKSkgewogIGRpci5jcmVhdGUob3V0cHV0X2RpciwgcmVjdXJzaXZlID0gVFJVRSkKfQpgYGAKCiMjIyMgSW5wdXQKCldlIGhhdmUgVlNUIHRyYW5zZm9ybWVkIFJOQS1zZXEgZGF0YSwgYW5ub3RhdGVkIHdpdGggZ2VuZSBzeW1ib2xzLCB0aGF0IGhhcyBiZWVuIGNvbGxhcHNlZCBzdWNoIHRoYXQgdGhlcmUgYXJlIG5vIGR1cGxpY2F0ZWQgZ2VuZSBpZGVudGlmaWVycyAoc2VlIGBzZXR1cGApLgoKYGBge3IgaW5wdXRfZmlsZX0Kcm5hc2VxX2ZpbGUgPC0gZmlsZS5wYXRoKGlucHV0X2RpciwgIm1lZHVsbG9ibGFzdG9tYV92c3RfY29sbGFwc2VkLnRzdiIpCmBgYAoKIyMjIyBPdXRwdXQKCmBgYHtyIG91dHB1dF9maWxlLCBsaXZlID0gVFJVRX0KZ3N2YV9yZXN1bHRzX2ZpbGUgPC0gZmlsZS5wYXRoKG91dHB1dF9kaXIsICJtZWR1bGxvYmxhc3RvbWFfZ3N2YV9yZXN1bHRzLnRzdiIpCmBgYAoKIyMgR2VuZSBzZXRzCgpUaGUgZnVuY3Rpb24gdGhhdCB3ZSB3aWxsIHVzZSB0byBydW4gR1NWQSB3YW50cyB0aGUgZ2VuZSBzZXRzIHRvIGJlIGluIGEgbGlzdCwgcmF0aGVyIHRoYW4gYSB0aWR5IGRhdGEgZnJhbWUgdGhhdCB3ZSB1c2VkIHdpdGggYGNsdXN0ZXJQcm9maWxlcmAgKGFsdGhvdWdoIGl0IGRvZXMgYWNjZXB0IG90aGVyIGZvcm1hdHMpLgoKV2UncmUgZ29pbmcgdG8gdGFrZSB0aGlzIG9wcG9ydHVuaXR5IHRvIGludHJvZHVjZSBhIGRpZmZlcmVudCBmb3JtYXQgdGhhdCBnZW5lIHNldHMgYXJlIG9mdGVuIGRpc3RyaWJ1dGVkIGluIGNhbGxlZCBbR01UIChHZW5lIE1hdHJpeCBUcmFuc3Bvc2VkKV0oaHR0cHM6Ly9zb2Z0d2FyZS5icm9hZGluc3RpdHV0ZS5vcmcvY2FuY2VyL3NvZnR3YXJlL2dzZWEvd2lraS9pbmRleC5waHAvRGF0YV9mb3JtYXRzI0dNVDpfR2VuZV9NYXRyaXhfVHJhbnNwb3NlZF9maWxlX2Zvcm1hdF8uMjguMkEuZ210LjI5KS4KCldlJ3JlIGdvaW5nIHRvIHJlYWQgaW4gdGhlIEhhbGxtYXJrIGNvbGxlY3Rpb24gZmlsZSBkaXJlY3RseSBmcm9tIFtNU2lnREJdKGh0dHBzOi8vd3d3LmdzZWEtbXNpZ2RiLm9yZy9nc2VhL21zaWdkYi9pbmRleC5qc3ApLCByYXRoZXIgdGhhbiB1c2luZyBgbXNpZ2RicmAgbGlrZSB3ZSBkaWQgaW4gZWFybGllciBub3RlYm9va3MuCgpUaGUgUk5BLXNlcSBkYXRhIHVzZXMgZ2VuZSBzeW1ib2xzLCBzbyB3ZSBuZWVkIGdlbmUgc2V0cyB0aGF0IHVzZSBnZW5lIHN5bWJvbHMsIHRvby4KCmBgYHtyIGdtdH0KIyBSIGNhbiBvZnRlbiByZWFkIGluIGRhdGEgZnJvbSBhIFVSTApoYWxsbWFya3NfdXJsIDwtICJodHRwczovL2RhdGEuYnJvYWRpbnN0aXR1dGUub3JnL2dzZWEtbXNpZ2RiL21zaWdkYi9yZWxlYXNlLzcuMS9oLmFsbC52Ny4xLnN5bWJvbHMuZ210IgoKIyBRdVNBR0UgaXMgYW5vdGhlciBwYXRod2F5IGFuYWx5c2lzIG1ldGhvZCwgdGhlIHF1c2FnZSBwYWNrYWdlIGhhcyBhIGZ1bmN0aW9uCiMgZm9yIHJlYWRpbmcgR01UIGZpbGVzIGFuZCB0dXJuaW5nIHRoZW0gaW50byBhIGxpc3QKaGFsbG1hcmtzX2xpc3QgPC0gcXVzYWdlOjpyZWFkLmdtdChoYWxsbWFya3NfdXJsKQpgYGAKCldoYXQgZG9lcyB0aGlzIGxpc3QgbG9vayBsaWtlPwoKYGBge3IgaGVhZF9oYWxsbWFyaywgZXZhbCA9IEZBTFNFfQpoZWFkKGhhbGxtYXJrc19saXN0KQpgYGAKCiMjIFJOQS1zZXEgZGF0YQoKV2UgaGF2ZSBWU1QgdHJhbnNmb3JtZWQgUk5BLXNlcSBkYXRhLCB3aGljaCBpcyBvbiBhIGxvZzItbGlrZSBzY2FsZS4KVGhlc2UgZGF0YSBhcmUgZnJvbSB0aGUgT3BlbiBQZWRpYXRyaWMgQnJhaW4gVHVtb3IgQXRsYXMgKE9wZW5QQlRBKQpPcGVuUEJUQSBpcyBhIGNvbGxhYm9yYXRpdmUgcHJvamVjdCBvcmdhbml6ZWQgYnkgdGhlIENDREwgYW5kIHRoZSBDZW50ZXIgZm9yIERhdGEtRHJpdmVuIERpc2NvdmVyeSBpbiBCaW9tZWRpY2luZSAoRDNiKSBhdCB0aGUgQ2hpbGRyZW4ncyBIb3NwaXRhbCBvZiBQaGlsYWRlbHBoaWEgY29uZHVjdGVkIG9wZW5seSBvbiBHaXRIdWIuCgpZb3UgY2FuIHJlYWQgbW9yZSBhYm91dCB0aGUgcHJvamVjdCBbaGVyZV0oaHR0cHM6Ly9naXRodWIuY29tL2FsZXhzbGVtb25hZGUvb3BlbnBidGEtYW5hbHlzaXMvI29wZW5wYnRhLWFuYWx5c2lzKS4KCldlJ3JlIG9ubHkgd29ya2luZyB3aXRoIHRoZSBtZWR1bGxvYmxhc3RvbWEgc2FtcGxlcyBpbiB0aGlzIGV4YW1wbGUuCgpgYGB7ciByZWFkX2luX3JuYXNlcX0Kcm5hc2VxX2RmIDwtIHJlYWRyOjpyZWFkX3RzdihybmFzZXFfZmlsZSkKYGBgCgpgYGB7ciBwZWVrX3JuYXNlcSwgbGl2ZSA9IFRSVUV9CiMgV2hhdCBkb2VzIHRoZSBSTkEtc2VxIGRhdGEgZnJhbWUgbG9vayBsaWtlPwpybmFzZXFfZGZbMTo1LCAxOjVdCmBgYAoKRm9yIEdTVkEsIHdlIG5lZWQgYSBtYXRyaXguCgpgYGB7ciBybmFzZXFfbWF0LCBsaXZlID0gVFJVRX0Kcm5hc2VxX21hdCA8LSBybmFzZXFfZGYgJT4lCiAgdGliYmxlOjpjb2x1bW5fdG9fcm93bmFtZXMoImdlbmVfc3ltYm9sIikgJT4lCiAgYXMubWF0cml4KCkKYGBgCgoqTm90ZTogSWYgd2UgaGFkIGR1cGxpY2F0ZSBnZW5lIHN5bWJvbHMgaGVyZSwgd2UgY291bGRuJ3Qgc2V0IHRoZW0gYXMgcm93bmFtZXMuKgoKIyMgR1NWQQoKIVtdKGRpYWdyYW1zL2hhbnplbG1hbm5fZmlnMS5qcGcpCgoqKkZpZ3VyZSAxIGZyb20gW0jDpG56ZWxtYW5uIF9ldCBhbC5fICgyMDEzKV0oaHR0cHM6Ly9kb2kub3JnLzEwLjExODYvMTQ3MS0yMTA1LTE0LTcpLioqCgpZb3UgbWF5IG5vdGljZSB0aGF0IEdTVkEgaGFzIHNvbWUgY29tbW9uYWxpdGllcyB3aXRoIEdTRUEuClJhdGhlciB0aGFuIHJhbmtpbmcgZ2VuZXMgYmFzZWQgb24gc29tZSBzdGF0aXN0aWMgX3dlXyBzZWxlY3RlZCBhaGVhZCBvZiB0aW1lLCBHU1ZBIGZpdHMgYSBtb2RlbCBhbmQgcmFua3MgZ2VuZXMgYmFzZWQgb24gdGhlaXIgZXhwcmVzc2lvbiBsZXZlbCByZWxhdGl2ZSB0byB0aGUgc2FtcGxlIGRpc3RyaWJ1dGlvbi4KVGhpcyBpcyBhIHdheSBvZiBhc2tpbmcgaWYgYSBnZW5lIF9pXyBpcyBoaWdobHkgb3IgbG93bHkgZXhwcmVzc2VkIGluIGEgc2FtcGxlIF9qXyBpbiB0aGUgY29udGV4dCBvZiB0aGlzIGV4cGVyaW1lbnQgYW5kIHJhbmtpbmcgYWNjb3JkaW5nbHkgKFtIw6RuemVsbWFubiBfZXQgYWwuXyAyMDEzXShodHRwczovL2RvaS5vcmcvMTAuMTE4Ni8xNDcxLTIxMDUtMTQtNykpLgpUaGUgcGF0aHdheS1sZXZlbCBzY29yZSBjYWxjdWxhdGVkIGlzIGEgd2F5IG9mIGFza2luZyBob3cgZ2VuZXMgX3dpdGhpbl8gYSBnZW5lIHNldCB2YXJ5IGFzIGNvbXBhcmVkIHRvIGdlbmVzIHRoYXQgYXJlIF9vdXRzaWRlXyBvZiB0aGF0IGdlbmUgc2V0IChbTWFsaG90cmEuIDIwMThdKGh0dHBzOi8vdG93YXJkc2RhdGFzY2llbmNlLmNvbS9kZWNvZGluZy1nZW5lLXNldC12YXJpYXRpb24tYW5hbHlzaXMtODE5M2EwY2ZkYTMpKS4gCihUaGlzIGlzIHNvbWV0aW1lcyBjYWxsZWQgYSBjb21wZXRpdGl2ZSB0ZXN0IGluIGdlbmUgc2V0IGVucmljaG1lbnQgbGl0ZXJhdHVyZS4pClRoZSBpbnR1aXRpb24gaGVyZSBpcyB0aGF0IHdlIHdpbGwgZ2V0IHBhdGh3YXktbGV2ZWwgc2NvcmVzIGZvciBlYWNoIHNhbXBsZSB0aGF0IGluZGljYXRlIGlmIGdlbmVzIGluIGEgcGF0aHdheSB2YXJ5IGNvbmNvcmRhbnRseSBpbiBvbmUgZGlyZWN0aW9uIChvdmVyZXhwcmVzc2VkIG9yIHVuZGVyZXhwcmVzc2VkIHJlbGF0aXZlIHRvIHRoZSBvdmVyYWxsIHBvcHVsYXRpb24pIChbSMOkbnplbG1hbm4gX2V0IGFsLl8gMjAxM10oaHR0cHM6Ly9kb2kub3JnLzEwLjExODYvMTQ3MS0yMTA1LTE0LTcpKS4KClRoZSBvdXRwdXQgaXMgYSBnZW5lIHNldCBieSBzYW1wbGUgbWF0cml4IG9mIEdTVkEgc2NvcmVzLgoKIyMjIFBlcmZvcm0gR1NWQQoKYGBge3IgcnVuX2dzdmF9CmdzdmFfcmVzdWx0cyA8LSBnc3ZhKHJuYXNlcV9tYXQsIAogICAgICAgICAgICAgICAgICAgICBoYWxsbWFya3NfbGlzdCwKICAgICAgICAgICAgICAgICAgICAgbWV0aG9kID0gImdzdmEiLAogICAgICAgICAgICAgICAgICAgICAjIEFwcHJvcHJpYXRlIGZvciBvdXIgdHJhbnNmb3JtZWQgZGF0YSBvbiBsb2cyLWxpa2Ugc2NhbGUKICAgICAgICAgICAgICAgICAgICAga2NkZiA9ICJHYXVzc2lhbiIsICAKICAgICAgICAgICAgICAgICAgICAgIyBNaW5pbXVtIGdlbmUgc2V0IHNpemUKICAgICAgICAgICAgICAgICAgICAgbWluLnN6ID0gMTUsIAogICAgICAgICAgICAgICAgICAgICAjIE1heGltdW0gZ2VuZSBzZXQgc2l6ZQogICAgICAgICAgICAgICAgICAgICBtYXguc3ogPSA1MDAsCiAgICAgICAgICAgICAgICAgICAgICMgQ29tcHV0ZSBHYXVzc2lhbi1kaXN0cmlidXRlZCBzY29yZXMKICAgICAgICAgICAgICAgICAgICAgbXguZGlmZiA9IFRSVUUpCmBgYAoKKipOb3RlOiB0aGUgYGdzdmEoKWAgZG9jdW1lbnRhdGlvbiBzYXlzIHdlIGNhbiB1c2UgYGtjZGYgPSAiR2F1c3NpYW4iYCBpZiB3ZSBoYWQgUk5BLXNlcSBsb2ctQ1BNcywgbG9nLVJQS01zIG9yIGxvZy1UUE1zLCBidXQgd2Ugd291bGQgdXNlIGBrY2RmID0gIlBvaXNzb24iYCBvbiBpbnRlZ2VyIGNvdW50cy4qKgoKYGBge3IgZ3N2YV9wZWVrfQojIExldCdzIGV4cGxvcmUgd2hhdCB0aGUgb3V0cHV0IG9mIGdzdmEoKSBsb29rcyBsaWtlCmdzdmFfcmVzdWx0c1sxOjUsIDE6NV0KYGBgCgojIyMgQSBub3RlIG9uIGdlbmUgc2V0IHNpemUKIApPZnRlbiB0aGUgc2NvcmVzIG9mIGdlbmUgc2V0IGVucmljaG1lbnQgbWV0aG9kcyBhcmUgbm90IGNvbXBhcmFibGUgYmV0d2VlbiBnZW5lIHNldHMgb2YgZGlmZmVyZW50IHNpemVzLgpMZXQncyBkbyBhbiBleHBlcmltZW50IHVzaW5nIHJhbmRvbWx5IGdlbmVyYXRlZCBnZW5lIHNldHMgdG8gZXhwbG9yZSB0aGlzIGlkZWEgYSBiaXQgbW9yZS4KCldlIG5lZWQgdG8gZ2V0IGEgY29sbGVjdGlvbiBvZiBhbGwgcG9zc2libGUgZ2VuZXMgd2Ugd2lsbCBzYW1wbGUgZnJvbSB0byBjcmVhdGUgcmFuZG9tIGdlbmUgc2V0cy4KQmVjYXVzZSB3ZSdyZSBkb2luZyBzb21lIHJhbmRvbSBzYW1wbGluZywgd2UgbmVlZCB0byBzZXQgYSBzZWVkIGZvciB0aGlzIHRvIGJlIHJlcHJvZHVjaWJsZS4KCmBgYHtyIHJhbmRvbV9zZXR1cH0KIyBVc2UgYWxsIHRoZSBnZW5lIHN5bWJvbHMgaW4gdGhlIGRhdGFzZXQgYXMgdGhlIHBvb2wgb2YgcG9zc2libGUgZ2VuZXMKYWxsX2dlbmVzIDwtIHJvd25hbWVzKHJuYXNlcV9tYXQpCgojIFNldCBhIHNlZWQgZm9yIHJlcHJvZHVjaWJpbGl0eQpzZXQuc2VlZCgyMDIwKQpgYGAKCk91ciBtaW5pbXVtIGdlbmUgc2V0IHNpemUgZWFybGllciB3YXMgMTUgZ2VuZXMgYW5kIG91ciBtYXhpbXVtIGdlbmUgc2V0IHNpemUgd2FzIDUwMCBnZW5lcy4gCldlJ2xsIHVzZSB0aGUgc2FtZSBtaW5pbXVtIGFuZCBtYXhpbXVtIHZhbHVlcyBmb3Igb3VyIHJhbmRvbSBnZW5lIHNldHMgYW5kIHNvbWUgdmFsdWVzIGluIGJldHdlZW4uCgpgYGB7ciBnZW5lX3NldF9zaXplc30KIyBNYWtlIGEgbGlzdCBvZiBpbnRlZ2VycyB0aGF0IGluZGljYXRlIHRoZSByYW5kb20gZ2VuZSBzZXQgc2l6ZXMKZ2VuZV9zZXRfc2l6ZSA8LSBsaXN0KDE1LCAyNSwgNTAsIDEwMCwgMjUwLCA1MDApCmBgYAoKRm9yIGVhY2ggZ2VuZSBzZXQgc2l6ZSwgd2Ugd2lsbCBnZW5lcmF0ZSAxMDAgcmFuZG9tIGdlbmUgc2V0cy4KCmBgYHtyIHJhbmRvbV9nZW5lX3NldHN9CiMgU2V0IG51bWJlciBvZiByZXBsaWNhdGVzCm5yZXBzIDwtIDEwMAojIEdlbmVyYXRlIDEwMCByYW5kb20gZ2VuZSBzZXRzIG9mIGVhY2ggc2l6ZQpyYW5kb21fZ2VuZV9zZXRzIDwtIAogIHB1cnJyOjptYXAoCiAgICByZXAoZ2VuZV9zZXRfc2l6ZSwgbnJlcHMpLCAgIyBSZXBlYXQgZ2VuZSBzaXplcyBzbyB3ZSBydW4gYG5yZXBzYCB0aW1lcyAKICAgICMgU2FtcGxlIHRoZSB2ZWN0b3Igb2YgYWxsIGdlbmVzLCBjaG9vc2luZyB0aGUgbnVtYmVyIG9mIGl0ZW1zIHNwZWNpZmllZAogICAgIyBpbiB0aGUgZWxlbWVudCBvZiBnZW5lIHNldCBzaXplCiAgICB+IGJhc2U6OnNhbXBsZSh4ID0gYWxsX2dlbmVzLAogICAgICAgICAgICAgICAgICAgc2l6ZSA9IC54KQogICkKYGBgCgpUaGUgSGFsbG1hcmtzIGxpc3Qgd2UgdXNlZCBlYXJsaWVyIHN0b3JlZCB0aGUgZ2VuZSBzZXQgbmFtZXMgYXMgdGhlIG5hbWUgb2YgdGhlIGxpc3QsIHNvIGxldCdzIGFkZCBuYW1lcyB0byBvdXIgcmFuZG9tIGdlbmUgc2V0cyB0aGF0IGluZGljYXRlIHdoYXQgc2l6ZSB0aGV5IGFyZSBhbmQgc28gYGdzdmEoKWAgZG9lc24ndCBnZXQgdXBzZXQuCgpgYGB7ciBuYW1lX3JhbmRvbV9nZW5lX3NldHN9CiMgV2Ugd2lsbCBpbmNsdWRlIHRoZSBzaXplIG9mIHRoZSBnZW5lIHNldCBpbiB0aGUgZ2VuZSBzZXQgbmFtZSAKIyBTdGFydCBieSB0YWtpbmcgdGhlIGxlbmd0aCBvZiBlYWNoIHBhdGh3YXkgYW5kIGFwcGVuZGluZyAicGF0aHdheV8iIHRvIHRoYXQKIyBudW1iZXIKbGVuZ3Roc192ZWN0b3IgPC0gcmFuZG9tX2dlbmVfc2V0cyAlPiUKICAjIEdldCB0aGUgbGVuZ3RoIG9mIGVhY2ggZ2VuZSBzZXQgKG51bWJlciBvZiBnZW5lcykKICBwdXJycjo6bWFwKH4gbGVuZ3RoKC54KSkgJT4lCiAgIyBNYWtlIGl0ICJwYXRod2F5XzxnZW5lIHNldCBzaXplPiIKICBwdXJycjo6bWFwKH4gcGFzdGUwKCJwYXRod2F5XyIsIC54KSkgJT4lCiAgIyBSZXR1cm4gYSB2ZWN0b3IKICBwdXJycjo6ZmxhdHRlbl9jaHIoKQoKIyBBZGQgdGhlIG5hbWVzIGluIGxlbmd0aHNfdmVjdG9yIHRvIHRoZSBsaXN0IC0gInBhdGh3YXlfPGdlbmUgc2V0IHNpemU+IgpyYW5kb21fZ2VuZV9zZXRzIDwtIHJhbmRvbV9nZW5lX3NldHMgJT4lIAogICMgbWFrZS5uYW1lcygpIGFwcGVuZHMgYSAidmVyc2lvbiIgaWYgc29tZXRoaW5nIGlzIG5vdCB1bmlxdWUKICBwdXJycjo6c2V0X25hbWVzKG5tID0gbWFrZS5uYW1lcyhsZW5ndGhzX3ZlY3RvciwgdW5pcXVlID0gVFJVRSkpCmBgYAogClJ1biBHU1ZBIG9uIG91ciBkYXRhc2V0IHdpdGggdGhlIHNhbWUgcGFyYW1ldGVycyBhcyBiZWZvcmUsIGJ1dCBub3cgd2l0aCByYW5kb20gZ2VuZSBzZXRzLgoKYGBge3IgcmFuZG9tX2dzdmEsIGxpdmUgPSBUUlVFfQpyYW5kb21fZ3N2YV9yZXN1bHRzIDwtIGdzdmEocm5hc2VxX21hdCwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICByYW5kb21fZ2VuZV9zZXRzLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgbWV0aG9kID0gImdzdmEiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIyBBcHByb3ByaWF0ZSBmb3Igb3VyIHRyYW5zZm9ybWVkIGRhdGEgb24gCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAjIGxvZzItbGlrZSBzY2FsZQogICAgICAgICAgICAgICAgICAgICAgICAgICAga2NkZiA9ICJHYXVzc2lhbiIsICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgTWluaW11bSBnZW5lIHNldCBzaXplCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBtaW4uc3ogPSAxNSwgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAjIE1heGltdW0gZ2VuZSBzZXQgc2l6ZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgbWF4LnN6ID0gNTAwLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIyBDb21wdXRlIEdhdXNzaWFuLWRpc3RyaWJ1dGVkIHNjb3JlcwogICAgICAgICAgICAgICAgICAgICAgICAgICAgbXguZGlmZiA9IFRSVUUpICAgICAgICAgICAgICAgICAgICAgICAgICAgIApgYGAKCk5vdyBsZXQncyBtYWtlIGEgcGxvdCB0byBsb29rIGF0IHRoZSBkaXN0cmlidXRpb24gb2Ygc2NvcmVzIGZyb20gcmFuZG9tIGdlbmUgc2V0cy4KRmlyc3Qgd2UgbmVlZCB0byBnZXQgdGhpcyBkYXRhIGluIGFuIGFwcHJvcHJpYXRlIGZvcm1hdCBmb3IgYGdncGxvdDJgLgoKYGBge3IgbG9uZ2VyX3JhbmRvbV9nc3ZhfQojIFRoZSByYW5kb20gcmVzdWx0cyBhcmUgYSBtYXRyaXgKcmFuZG9tX2xvbmdfZGYgPC0gcmFuZG9tX2dzdmFfcmVzdWx0cyAlPiUKICBkYXRhLmZyYW1lKCkgJT4lCiAgIyBHZW5lIHNldCBuYW1lcyBhcmUgcm93bmFtZXMKICB0aWJibGU6OnJvd25hbWVzX3RvX2NvbHVtbigiZ2VuZV9zZXQiKSAlPiUKICAjIEdldCBpbnRvIGxvbmcgZm9ybWF0CiAgdGlkeXI6OnBpdm90X2xvbmdlcihjb2xzID0gLWdlbmVfc2V0LCAKICAgICAgICAgICAgICAgICAgICAgIG5hbWVzX3RvID0gIktpZHNfRmlyc3RfQmlvc3BlY2ltZW5fSUQiLAogICAgICAgICAgICAgICAgICAgICAgdmFsdWVzX3RvID0gImdzdmFfc2NvcmUiKSAlPiUKICAjIFJlbW92ZSB0aGUgLnZlcnNpb24gYWRkZWQgYnkgbWFrZS5uYW1lcygpCiAgZHBseXI6Om11dGF0ZShnZW5lX3NldCA9IHN0cmluZ3I6OnN0cl9yZW1vdmUoZ2VuZV9zZXQsICJcXC4uKiIpKSAlPiUKICAjIEFkZCBhIGNvbHVtbiB0aGF0IGtlZXBzIHRyYWNrIG9mIHRoZSBnZW5lIHNldCBzaXplCiAgZHBseXI6Om11dGF0ZShnZW5lX3NldF9zaXplID0gc3RyaW5ncjo6d29yZChnZW5lX3NldCwgMiwgc2VwID0gIl8iKSkgJT4lCiAgIyBXZSB3YW50IHRvIHBsb3Qgc21hbGxlc3Qgbm8uIGdlbmVzIC0+IGxhcmdlc3Qgbm8uIGdlbmVzCiAgZHBseXI6Om11dGF0ZShnZW5lX3NldF9zaXplID0gZmFjdG9yKGdlbmVfc2V0X3NpemUsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBsZXZlbHMgPSBjKDE1LCAyNSwgNTAsIDEwMCwgMjUwLCA1MDApKSkKCmBgYAoKTGV0J3MgbWFrZSBhIHZpb2xpbiBwbG90IHNvIHdlIGNhbiBsb29rIGF0IHRoZSBkaXN0cmlidXRpb24gb2Ygc2NvcmVzIGJ5IGdlbmUgc2V0IHNpemUuCgpgYGB7ciByYW5kb21fdmlvbGlufQojIFZpb2xpbiBwbG90IGNvbXBhcmluZyBHU1ZBIHNjb3JlcyBvZiBkaWZmZXJlbnQgcmFuZG9tIGdlbmUgc2V0IHNpemVzCnJhbmRvbV9sb25nX2RmICU+JQogIGdncGxvdDI6OmdncGxvdChnZ3Bsb3QyOjphZXMoeCA9IGdlbmVfc2V0X3NpemUsIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgeSA9IGdzdmFfc2NvcmUpKSArCiAgIyBNYWtlIGEgdmlvbGluIHBsb3QgdGhhdCdzIGEgcHJldHR5IGJsdWUhCiAgZ2dwbG90Mjo6Z2VvbV92aW9saW4oZmlsbCA9ICIjOTlDQ0ZGIiwgYWxwaGEgPSAwLjUpICsKICAjIEFkZCBhIHBvaW50IHdpdGggdGhlIG1lYW4gdmFsdWUKICBnZ3Bsb3QyOjpzdGF0X3N1bW1hcnkoCiAgICBnZW9tID0gInBvaW50IiwKICAgIGZ1biA9ICJtZWFuIiwKICAgICMgQ2hhbmdlIHRoZSBhZXN0aGV0aWNzIG9mIHRoZSBwb2ludHMKICAgIHNpemUgPSAzLAogICAgY29sb3IgPSAiIzAwNjZDQyIsCiAgICBzaGFwZSA9IDE4CiAgKSArCiAgIyBGbGlwIHRoZSBheGVzCiAgZ2dwbG90Mjo6Y29vcmRfZmxpcCgpICsKICBnZ3Bsb3QyOjpsYWJzKHRpdGxlID0gIlJhbmRvbSBHZW5lIFNldCBHU1ZBIFNjb3JlcyIsCiAgICAgICAgICAgICAgICB4ID0gImdlbmUgc2V0IHNpemUiLAogICAgICAgICAgICAgICAgeSA9ICJHU1ZBIHNjb3JlIikgKwogIGdncGxvdDI6OnRoZW1lX2J3KCkKYGBgCgpXaGF0IGRvIHlvdSBub3RpY2UgYWJvdXQgdGhlc2UgZGlzdHJpYnV0aW9ucz8KSG93IG1pZ2h0IHlvdSB1c2UgdGhpcyBpbmZvcm1hdGlvbiB0byBpbmZvcm0geW91ciBpbnRlcnByZXRhdGlvbiBvZiBHU1ZBIHNjb3Jlcz8KCiMjIyBIb3cgY2FuIHlvdSB1c2UgdGhlc2Ugc2NvcmVzPwoKSWYgeW91IGRpZCBoYXZlIGdyb3VwcyBpbmZvcm1hdGlvbiBmb3IgeW91ciBzYW1wbGVzLCB5b3UgY291bGQgdGVzdCBmb3IgcGF0aHdheSBzY29yZXMgZGlmZmVyZW5jZXMgYmV0d2VlbiBncm91cHMuCkhlcmUncyBbYW4gZXhhbXBsZV0oaHR0cHM6Ly9odG1scHJldmlldy5naXRodWIuaW8vP2h0dHBzOi8vZ2l0aHViLmNvbS9BbGV4c0xlbW9uYWRlL09wZW5QQlRBLWFuYWx5c2lzL2Jsb2IvOWI0NGJmMWMxODZiMzEyNmIxNmRiZTViODc3NTZiM2VhZTNmZWVjMi9hbmFseXNlcy9nZW5lLXNldC1lbnJpY2htZW50LWFuYWx5c2lzLzAyLW1vZGVsLWdzZWEubmIuaHRtbCkgb2YgdGhhdCBmcm9tIHRoZSBPcGVuUEJUQSBwcm9qZWN0IGl0c2VsZiEgCgpZb3UgY2FuIGFsc28gdmlzdWFsaXplIHRoaXMgbWF0cml4IGluIGEgaGVhdG1hcC4KSGVyZSdzIGEgZmlndXJlIGZyb20gdGhlIE9wZW5QQlRBIHByb2plY3QsIHdoZXJlIHRoZSBtaWRkbGUgcGFuZWwgaXMgYSBoZWF0bWFwIG9mIEdTVkEgc2NvcmVzIHRoYXQgd2VyZSBzaWduaWZpY2FudGx5IGRpZmZlcmVudCBiZXR3ZWVuIGhpc3RvbG9naWVzLgoKIVtdKGh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9BbGV4c0xlbW9uYWRlL09wZW5QQlRBLWFuYWx5c2lzLzliNDRiZjFjMTg2YjMxMjZiMTZkYmU1Yjg3NzU2YjNlYWUzZmVlYzIvZmlndXJlcy9wbmdzL3RyYW5zY3JpcHRvbWljLW92ZXJ2aWV3LnBuZykKClsqQ29kZSBoZXJlLipdKGh0dHBzOi8vZ2l0aHViLmNvbS9BbGV4c0xlbW9uYWRlL09wZW5QQlRBLWFuYWx5c2lzL2Jsb2IvOWI0NGJmMWMxODZiMzEyNmIxNmRiZTViODc3NTZiM2VhZTNmZWVjMi9maWd1cmVzL3NjcmlwdHMvdHJhbnNjcmlwdG9taWMtb3ZlcnZpZXcuUiNMMTA2KQoKIyMjIFdyaXRlIHJlc3VsdHMgdG8gZmlsZQoKYGBge3Igd3JpdGVfZ3N2YV9yZXN1bHRzfQpnc3ZhX3Jlc3VsdHMgJT4lCiAgYXMuZGF0YS5mcmFtZSgpICU+JQogIHRpYmJsZTo6cm93bmFtZXNfdG9fY29sdW1uKCJwYXRod2F5IikgJT4lCiAgcmVhZHI6OndyaXRlX3RzdihmaWxlID0gZ3N2YV9yZXN1bHRzX2ZpbGUpCmBgYAoKIyMgU2Vzc2lvbiBJbmZvCgpgYGB7ciBzZXNzaW9uX2luZm99CnNlc3Npb25JbmZvKCkKYGBgCg==